Establish an evidence-bounded content-readiness baseline - #1
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR reframes aeoptimize into an evidence-bounded, deterministic content-readiness linter (rather than an AI-citation/ranking predictor), and establishes a reproducible v0.6 baseline across rules, docs, packaging, and CI/Action surfaces.
Changes:
- Reworked scoring methodology/rules to remove speculative signals (FAQ/llms.txt) and flag unsourced quantitative claims; tightened schema generation boundaries.
- Updated CLI/plugin messaging + documentation to align with “content readiness” framing and explicit evidence boundaries.
- Added/updated GitHub Action + CI contract tests; bumped Node support and refreshed dependencies/lockfile.
Reviewed changes
Copilot reviewed 35 out of 36 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/plugins/vite.ts | Update plugin console output to “readiness score” + clarify artifacts are optional |
| src/plugins/next.ts | Update plugin console output to “readiness score” + clarify artifacts are optional |
| src/core/scanner.ts | Remove llms.txt HEAD discovery and rename summary framing |
| src/core/rules.ts | Rebalance weights + remove scoring for FAQ/llms.txt; adjust schema/meta/quantitative-claim heuristics |
| src/core/merger.ts | Clarify AI blend methodology wording as experimental/non-predictive |
| src/core/generator.ts | Tighten schema inference (Article requires stronger signals); stop FAQ inference; update crawler copy |
| src/core/ai-prompt.ts | Rewrite AI prompt to emphasize evidence boundaries and non-predictive scoring |
| src/core/tests/scanner.test.ts | Update fixtures expectations; adjust scanUrl redirect test for removed llms.txt fetch |
| src/core/tests/rules.test.ts | Update expectations for “JSON-LD is optional” behavior and content-readiness wording |
| src/core/tests/generator.test.ts | Update schema generation tests (no FAQ inference; stricter Article conditions) |
| src/core/tests/fixtures/good-page.md | Rewrite fixture to remove fabricated outcome claims and reflect evidence-bounded language |
| src/core/tests/fixtures/good-page.html | Rewrite fixture to remove fabricated stats, update schema + copy to evidence framing |
| src/core/tests/evidence-boundaries.test.ts | Add tests asserting key “evidence boundary” behaviors |
| src/cli/index.ts | Version bump to 0.6.0 + CLI messaging updates + output wording changes |
| skills/aeo-transform/SKILL.md | Update skill guidance to forbid invented claims/outcome language; define allowed transformations |
| skills/aeo-scan/SKILL.md | Update skill guidance to deterministic readiness framing; constrain experimental AI review |
| skills/aeo-generate/SKILL.md | Update generator skill guidance (preview-first, no llms-txt rel, no FAQ inference) |
| SECURITY.md | Add security policy + reporting guidance |
| ROADMAP.md | Add v0.6 evidence baseline roadmap + explicit non-goals |
| README.md | Rewrite README to deterministic readiness framing; update examples and CI/action guidance |
| package.json | Bump to 0.6.0; add docs files to package; raise Node engine to >=22.12.0; dependency bumps |
| package-lock.json | Refresh lockfile for updated deps and Node constraints |
| docs/superpowers/specs/2026-04-07-framework-plugins-design.md | Mark as historical, superseded by v0.6 methodology |
| docs/methodology.md | Add v0.6 methodology + evidence classes + exclusions/safety policy |
| CONTRIBUTING.md | Add contribution requirements around evidence, fixtures, and reproducibility |
| CHANGELOG.md | Add 0.6.0 changelog entry capturing the reframing and contract changes |
| agents/aeo-analyzer.md | Update agent instructions to evidence-bounded review framing |
| agents/aeo-ai-scorer.md | Update scorer instructions to explicitly experimental/non-predictive output |
| action/test-contract.sh | Add Action contract test script for advisory/blocking modes + invalid input handling |
| action/run.sh | Implement Action runtime script: validate inputs, run scan, emit outputs, enforce threshold |
| action/action.yml | Provide compatibility composite action wrapper for older .../action usage |
| action.yml | Add root-level composite action entrypoint |
| .github/workflows/ci.yml | Update CI matrix to Node 22/24; add action contract job; add pack/audit steps |
| .github/fixtures/action-low/index.html | Add minimal fixture for Action contract tests |
| .claude-plugin/plugin.json | Update plugin metadata to v0.6 framing and repo identity |
| .claude-plugin/marketplace.json | Update marketplace metadata to v0.6 framing and repo identity |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
54
to
55
| const existing = new Set(doc.jsonLd.map((ld) => ld['@type']).filter(Boolean)); | ||
| const generated: object[] = []; |
Comment on lines
+31
to
+33
|
|
||
| REPORT=$(run_aeoptimize scan "$INPUT_PATH" --dir --json 2>/dev/null) | ||
| SCORE=$(printf '%s' "$REPORT" | node -e "const fs=require('node:fs');const r=JSON.parse(fs.readFileSync(0,'utf8'));const s=r?.overall?.total;if(!Number.isInteger(s)||s<0||s>100)process.exit(2);console.log(s)") |
Comment on lines
+46
to
+48
| expect(result.score).toBe(result.maxScore); | ||
| expect(result.issues.join(' ')).not.toContain('exactly one H1'); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
llms.txtand FAQ presence non-scoring, remove rigid meta-description length and multiple-H1 penalties, and flag unsourced quantitative claims.action.yml, refresh package metadata, and update CI to supported Node.js 22 and 24 releases.Why
The previous scoring and documentation mixed machine-checkable delivery signals with claims that cannot be validated from a static page scan. The repository also tested Node.js 18 even though its current dependency graph required newer runtime APIs.
This change establishes explicit evidence boundaries and a reproducible maintenance baseline before public promotion or adoption work.
Impact
Existing CLI and plugin entry points remain available. Some scores may decrease or change because speculative signals no longer receive weight. Consumers should treat the output as a readiness report, not a prediction of search ranking or AI citations.
Validation
npm pack --dry-runpassednpm audit --audit-level=high: 0 vulnerabilitiesgit diff --checkpassed